Stabilizing Motion Of An Interaction Ray

ABSTRACT

Technology for stabilizing an interaction ray based on variance in head rotation is disclosed. One aspect includes monitoring orientation of a person&#39;s head, which may include monitoring rotation about an axis of the head, such as recording an Euler angle with respect to rotation about an axis of the head. The logic determines a three-dimensional (3D) ray based on the orientation of the head. The 3D ray has a motion that precisely tracks the Euler angle over time. The logic generates an interaction ray that tracks the 3D ray to some extent. The logic determines a variance of the Euler angle over time. The logic stabilizes the interaction ray based on the variance of the Euler angle over time despite some rotation about the axis of the head. The amount of stabilizing may be inversely proportional to the variance of the Euler angle.

CLAIM OF PRIORITY

This application is a continuation application of U.S. patentapplication Ser. No. 14/485,542 filed on Sep. 12, 2014 and published asUS 2016/0077344 on Mar. 17, 2016, entitled “STABILIZING MOTION OF ANINTERACTION RAY.”

BACKGROUND

Numerous techniques have been suggested for allowing a user to point ora select when using an electronic device. For example, a user cancontrol a cursor on a display screen to select, scroll, etc. Suchcursors could be controlled by a computer mouse, a trackball, a touchpadetc. Some devices have touchscreens for user input. More recently,techniques that employ eye tracking or head tracking have been suggestedto allow user input, selection, etc.

SUMMARY

Embodiments of the present technology relate to a system, device, andmethod for stabilizing an interaction ray based on variance in angle ofhead rotation.

One embodiment includes an apparatus having a sensor and logic thatmonitors orientation of a person's head using the sensor. Thismonitoring may include monitoring rotation about an axis of the head,such as recording an Euler angle with respect to rotation about the axisof the head. The logic determines a three-dimensional (3D) ray based onthe orientation of the head. The 3D ray has a motion that preciselytracks the Euler angle over time. The logic generates an interaction raythat tracks the 3D ray to some extent. The logic determines a varianceof the Euler angle over time. The logic stabilizes the interaction raybased on the variance of the Euler angle over time despite some rotationabout the axis of the head. The amount of stabilizing is inverselyproportional to the variance of the Euler angle. The logic determines acollision of the second 3D ray with a 3D coordinate.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an example environment in which embodiments of controllingmotion of an interaction ray may be practiced.

FIG. 1B is a diagram of a person's head to help illustrate an embodimentthat monitors pitch, roll, and yaw.

FIG. 1C is a flowchart of one embodiment of a process of controllingmotion in an interaction ray.

FIG. 2 is a perspective view of one embodiment of a head mounted displayunit.

FIG. 3 is a side view of a portion of one embodiment of a head mounteddisplay unit.

FIG. 4 is a block diagram of one embodiment of the components of a headmounted display unit.

FIG. 5 is a block diagram of one embodiment of the components of aprocessing unit associated with a head mounted display unit.

FIG. 6A shows a 3D ray and an interaction ray for several points in timein which the interaction ray closely tracks the 3D ray.

FIGS. 6B and 6C show 3D rays for an example in which variance ofrotation about an axis is low and the interaction ray is stable.

FIG. 7 is a flowchart of one embodiment for determining an instabilityfactor based on variance in an Euler angle.

FIG. 8 is a flowchart of one embodiment of a process of applying aninstability factor to determine a new interaction ray.

FIG. 9 is a flowchart of one embodiment in which a head curser isstabilized based on variance of angle of rotation of the person's head.

FIG. 10 is a flowchart of one embodiment in which stabilizing isprovided for variance of translation of the person's head.

FIG. 11 is a flowchart showing that many combination of thestabilization factors can be applied to the interaction ray.

DETAILED DESCRIPTION

Embodiments disclosed herein provide for an interaction ray that can beused as a selector or pointer. For example, a user that is wearing ahead mounted display (HMD) may use the interaction ray to make aselection of an element being presented in the HMD. The user may controlthe interaction ray by orientation of their head. The interaction raymay be a 3D vector that originates from the user's head. Note that theinteraction ray is not necessarily a visible ray. The interaction rayserves as a type of cursor in one embodiment. As a particular example,the user may be reading a newspaper article being presented in the HMD.The interaction ray could allow the user to select or point to anelement such as a hyperlink in the article.

It could potentially be difficult for the user to control the locationof the interaction ray. However, embodiments disclosed herein controlmotion of a 3D ray that is calculated based on orientation of the user'shead in a way that allows precise control of the interaction ray. In oneembodiment, a high degree of stabilizing of the interaction ray isprovided when variance of head motion is low. This means that small headmovements are in effect stabilized, which provides more precise controlof the interaction ray. However, little or no stabilizing of theinteraction ray is provided when variance of head motion is high. Forexample, if the user is moving their head from right to left, littlestabilizing of the interaction ray is provided. This makes theinteraction ray more responsive to head motion, which means that theinteraction ray tracks the actual orientation of the user's head moreclosely.

FIG. 1A shows an example of a user 18 interacting with a virtual image60 by use of an interaction ray 66. The user 18 is wearing an HMD 2,which is displaying the virtual image 60. Thus, the location of thevirtual image 60 in FIG. 1A is meant to represent the illusion that theuser gets that the virtual image 60 is located somewhere in front of theuser 18. This may be referred to as the field of view of the HMD 2. Inthis example, the virtual image 60 may stay relatively fixed as the user18 moves their head 7 and/or eyes. Thus, the user can easily shift theirattention to a different location in the virtual image 60 withoutcausing the virtual image 60 to shift. The virtual image 60 could becontain content such as a virtual newspaper, as one example.

The interaction ray 66 originates from a point at or near the user'shead 7, such as midway between the eyes, in this example. Theinteraction ray 66 may be a 3D vector. The interaction ray 66 does notneed to be shown in the HMD 2, but that is one possibility. The user 18may move their head 7 to cause the interaction ray 66 to move. Forexample, as the user 18 moves his head 7 from right to left, theinteraction ray 66 tracks the orientation of the head 7.

One possible use of the interaction ray 66 is as a cursor. For example,the virtual image 60 could have some selectable elements 62, which mightbe links to other content. The user 18 might select one of theselectable elements 62 by positioning their head such that theinteraction ray 66 is pointing at the desired selectable element 62. Asnoted, the interaction ray 66 itself does not need to be visibly presentin the display. The HMD 2 might highlight the element 62 presently beingpointed to by the interaction ray. The user 18 might select the element62 in a number of ways such as a voice command, tapping a button or somephysical contact on the HMD 2, etc.

As noted above, one of the challenges of the interaction ray 66 is thatthe user could potentially have a difficult time precisely controllingthe location of the interaction ray. For example, slight head movementscould potentially cause undesired movements of the interaction ray 66.Embodiments stabilize the interaction ray 66 that is created based onmovement of the user's head 7 in a way to provide better control overthe interaction ray 66. In one embodiment, the stabilizing of theinteraction ray 66 is inversely proportional to the variance of headrotation. Further details of this will be discussed below.

FIG. 1A shows one possible HMD-based coordinate system. The origin ofthe HMD-based coordinate system is somewhere near the middle of theuser's head 7 in one embodiment. Another location for the origin ispossible. Thus, the location of the origin of the HMD-based coordinatesystem that is depicted in FIG. 1A should be understood to be for thesake of ease of illustration. The origin is not limited to the depictedlocation. In one embodiment, the original of the HMD-based coordinatesystem moves as the user moves their head 7, such that the HMD-basedcoordinate system stays fixed relative to the position of the HMD 2. Forexample, as the user's head translates laterally, the origin of theHMD-based coordinate system may translate an equal amount. However,rotations of the user's head about one of the axes do not move theHMD-based coordinate system, in one embodiment. For example, when theuser moves their head from right to left (such that their head rotatesabout the z-axis) the head movement can be measured relative to theHMD-based coordinate system in terms of angles of rotation with respectto the z-axis. Note that translations of the user's head may also betaken into consideration.

In one embodiment, one or more Euler angles are tracked. Euler anglesmay represent rotations about the axes of a coordinate system. Forexample, the angles may represent a rotation about the x-axis by anangle α, a rotation about the y-axis by an angle β, and a rotation aboutz-axis, by an angle γ. For example, one or more of pitch (y-axis), yaw(z-axis) and/or roll (x-axis) could be tracked.

FIG. 1B is a diagram of a person's head to help illustrate an embodimentthat monitors pitch, roll, and yaw. An example z-axis, y-axis, andx-axis with respect to the person's head 7. This coordinate system hasan origin somewhere in the person's head 7. Note that this may be thesame coordinate system as the HMD-based coordinate system from FIG. 1A.

The y-axis in FIG. 1B roughly corresponds to a line between the person'sears. The x-axis in FIG. 1B roughly corresponds to a line from a pointbetween the person's eyes to out through the back of the head. Thez-axis in FIG. 1B roughly corresponds to a line from the center of thehead upwards through the top of the head. These axes are shown for thesake of one example.

The following is an example usage scenario. A user may be sitting still,interacting with a horizontally laid out menu being presented in an HMD2. The variance of the head position may be low since the person isstationary. Therefore, subtle translation effects may be stabilized inone embodiment. The variance in pitch may be very low since the personmay be looking left to right, but not up and down. Therefore, a highdegree of stabilizing may be applied so the interaction ray will notmove up and down on the page. The volatility in yaw may be high sincethe person is rotating their head with respect to the z-axis as theylook along the list. Therefore, the stabilizing with respect to thez-axis rotation may be low to allow the interaction ray 66 to closelytrack this left to right movement of the person's head. However, if theuser slows their left to right (yaw) motion as they close in on a menuitem of interest, the variance drops. In response, the amount of thestabilization of the interaction ray 66 may be increased, allowing moreprecise selection.

Because the user might not know their head direction and orientationprecisely, it can be difficult for the user to precisely control tointeraction ray 66. However, embodiments stabilize the interaction ray66 and as such provide higher fidelity in refined motion.

FIG. 1C is a flowchart of one embodiment of a process of stabilizingmotion of an interaction ray 66. The process can be practiced in anenvironment of such as the one of FIG. 1A, but that is just one example.The process is performed by logic in an HMD 2, in one embodiment. Thelogic could be a processor that executes processor readableinstructions, hardware such as an application specific circuit (ASIC),System-on-a-Chip systems (SoCs), etc. Thus, the process can be performedby software (e.g., instructions that are stored on a storage device andexecuted by a processor), hardware, or some combination of software andhardware.

In step 42, the orientation of a person's head is tracked using asensor. Step 42 may include tracking rotation about one or more axes ofthe head. The three axes are an x-, y-, and z-axis in one embodiment. Inone embodiment, one or more Euler angles are tracked. Euler angles mayrepresent rotations about the axes of a coordinate system. For example,the angles may represent a rotation about the x-axis by an angle α, arotation about the y-axis by an angle β, and a rotation about z-axis, byan angle γ. For example, one or more of pitch (y-axis), yaw (z-axis)and/or roll (x-axis) could be tracked.

In step 44, a first 3D ray is determined based on actual orientation ofthe head. In one embodiment, the origin of this first 3D ray is at ornear the user's head. For example, the origin may be midway between theuser's eyes. The origin could be the same as the origin of the HMD-basedcoordinate system. This first 3D ray may extend outward in a directionin which the user is looking. However, it is not required that eyetracking be used. Rather, the direction of the first 3D ray may bedetermined entirely on the orientation of the user's head, in oneembodiment.

This first 3D ray may be the direct result of the rotation about the oneor more axis. For example, rotation about the z-axis of the head maydirectly result in a first component of motion of the first 3D vector.Likewise, rotation about the y-axis of the head may directly result in asecond component of motion of the first 3D vector. Furthermore, rotationabout the x-axis of the head may directly result in a third component ofmotion of the first 3D vector.

The first 3D ray may track the Euler angle over time. Note that this maybe a precise or faithful tracking in that direction of the first 3D raymay exactly correspond to whatever Euler angles are being monitored.One, two, or three Euler angles may be monitored. In one embodiment, thefirst 3D ray is simply the x-axis. In this case, the movement of thefirst 3D ray depends on pitch and yaw, but does not necessarily dependon roll. However, a roll component could be added to the first 3D vectorif desired.

The first 3D ray may be determined using sensors on an HMD 2. Thissensors could include cameras, accelerometers, etc. In one embodiment,image data is analyzed to determine both the user head position and aface unit vector looking straight out from a user's face. The face unitvector may be determined by defining a plane of the user's face, andtaking a vector perpendicular to that plane. This plane may beidentified by determining a position of a user's eyes, nose, mouth, earsor other facial features.

In step 46, a variance of the rotation about the one or more axes of theperson's head is determined. The variance refers to the variance of theEuler angle referred to in step 42 over time, in one embodiment. Notethat the variance does not necessarily refer to a strict definition ofthe mathematical term “variance”. However, one possibility is for thevariance, as the term as used herein, to refer to the mathematical termvariance. Thus the term, “variance” as used herein with respect to thevariance in the Euler angle, pitch, roll, yaw, etc. includes, but is notlimited to, the mathematical term “variance.”

In step 48, a second 3D ray is generated based on the first 3D ray. Thissecond 3D ray may also be referred to as an interaction ray 66.Generating the second 3D ray may include determining an origin and adirection for the second 3D ray. In other words, a 3D vector may bedetermined. The origin of the second 3D may be the same as the origin ofthe first 3D vector. However, the direction may not exactly coincidewith that of the first 3D vector. This can help to stabilize theinteraction ray 66 to allow for better user control of the interactionray 66.

Note that the interaction ray 66 may track the first 3D ray, but this isnot necessarily a precise tracking. Stated another way, the interactionray 66 may track the first 3D ray to some extent. In one embodiment, thesystem alters how closely the interaction ray 66 tracks the first 3D raybased on the variance of the Euler angle over time. The closeness of thetracking may be proportional to the variance of the Euler angle over arecent time period. In one embodiment, the system stabilizes theinteraction ray 66 based on the variance of the Euler angle over time.The amount of stabilizing may be inversely proportional to the varianceof the Euler angle over a recent time period. Further details arediscussed below.

In step 50, the interaction ray 66 is stabilized based on the varianceof the first 3D ray. The stabilizing is inversely proportional to thevariance of the Euler angle over a recent time period, in oneembodiment. For example, if the user is moving their head slowly, thisshould result in low variance in angle of rotation of their head aboutan axis that may be roughly in line with their spine. For low variance,high stabilizing may be applied to the motion of the interaction ray 66.This results in the interaction ray 66 being stable despite some motionof the first 3D ray. Referring to the example of FIG. 1A, this helps theuser to keep the interaction ray 66 stable despite small head movementsdue to, for example, breathing, etc.

However, for high variance of the Euler angle over a recent time period,low or no stabilizing would be applied to the motion of the interactionray, for one embodiment. This may result in the interaction ray 66tracking the first 3D ray very closely (possibly identically). Referringto the example of FIG. 1A, if the user were to move their head from leftto right, the interaction ray 66 may move without delay. This can helpthe interaction ray 66 to be highly responsive.

Note that when variance is determined for more than one axis, a separatestabilizing factor may be used in connection with rotation about eachaxis. For example, a separate stabilizing factor may be determined forone, two, or all three of the components of motion of the first 3Dvector discussed above.

In step 52, a collision between the second 3D ray (or interaction ray)and a 3D coordinate is determined. The 3D coordinate is a real worldcoordinate, in one embodiment. In the example of FIG. 1A, theinteraction ray 66 is depicted as colliding with one of the selectableelements 62. Thus, the system may detect an element 62 or some portionof the image 60 with which the interaction ray 66 collides. Here,“collides” is referring to the fact that the second 3D ray may beconsidered to occupy various points in 3D space. The 3D coordinate ofthe image 60 may be a 3D coordinate at which the image 60 appears to beat. Note that mixed reality techniques are well known in the art, andmay be one way of making the image 60 appear to be at some real world 3Dcoordinate. That is, in reality the image 60 may be physically presentedon the HMD 2. However, the 3D coordinates of the HMD display are notnecessarily the 3D coordinates being referred to in step 52.

The 3D coordinate (or multiple 3D coordinates) are not required to beassociated with image 60 being presented in the HMD 2. For example, the3D coordinates could be some point in the real world. Thus, the systemcould determine that the interaction ray 66 is pointed at some object inthe real world.

In various embodiments, the user wears a head mounted display deviceincluding a display element. Next, an example HMD system will bediscussed. The display element is to a degree transparent so that a usercan look through the display element at real world objects within theuser's field of view (FOV). The display element also provides theability to project virtual images into the FOV of the user such that thevirtual images may also appear alongside the real world objects. In oneembodiment, the system may automatically track where the user is lookingso that the system can determine where to insert the virtual image inthe FOV of the user. Once the system knows where to project the virtualimage, the image is projected using the display element. In oneembodiment, the system inserts the virtual image such that it appears toremain fixed at the same place in the real world. For example, thesystem can make it appear that a virtual newspaper remains in the sameplace as the user moves their head and/or eyes to read the virtualnewspaper.

In embodiments, the system builds a model of the environment includingthe x, y, z Cartesian positions of the user, real world objects andvirtual three-dimensional objects in the room or other environment. Thepositions of the head mounted display device worn by the user in theenvironment may be calibrated to the model of the environment. Thisallows the system to determine the user's line of sight and FOV of theenvironment. Note that a different coordinate system may be used for thepositions of the user, real world objects and virtual three-dimensionalobjects in the room or other environment than the previously mentionedHMD-based coordinate system. Appropriate translations can be madebetween the coordinate systems.

A user may choose to interact with one or more of the virtual objectsappearing within the user's FOV. The interaction ray 66 allows the userto specify a virtual object. A variety of techniques can be used toallow the user to select the virtual object. A user may interact withvirtual objects using verbal gestures, such as for example a spoken wordor phrase recognized by the mixed reality system as a user request forthe system to perform a predefined action. Verbal gestures may be usedin conjunction with physical gestures to interact with one or morevirtual objects in the mixed reality environment.

In accordance with the present technology, when multiple virtual objectsare displayed, the present system determines which of the virtualobjects the user is focused on. This may be based on the interaction ray66. That virtual object is then available for interaction and the othervirtual objects may, optionally, be deemphasized by various methods. Thepresent technology uses various schemes for determining user focus. Inone example, the system may receive a predefined selection gestureindicating that the user is selecting a given virtual object.Alternatively, the system may receive a predefined interaction gesture,where the user indicates a focus by interacting with a given virtualobject. Both the selection gesture and the interaction gestures may bephysical or verbal.

Embodiments are described below which identify user focus on a virtualobject such as a virtual display slate presenting content to a user. Thecontent may be any content which can be displayed on the virtual slate,including for example static content such as text and pictures ordynamic content such as video. However, it is understood that thepresent technology is not limited to identifying user focus on virtualdisplay slates, and may identify user focus on any virtual objects withwhich a user may interact.

As seen in FIG. 2, the head mounted display device 2 is in communicationwith its own processing unit 4 via wire 6. In other embodiments, headmounted display device 2 communicates with processing unit 4 viawireless communication. Head mounted display device 2, which in oneembodiment is in the shape of glasses, is worn on the head of a user sothat the user can see through a display and thereby have an actualdirect view of the space in front of the user. The use of the term“actual direct view” refers to the ability to see the real world objectsdirectly with the human eye, rather than seeing created imagerepresentations of the objects. For example, looking through glass at aroom allows a user to have an actual direct view of the room, whileviewing a video of a room on a television is not an actual direct viewof the room. More details of the head mounted display device 2 areprovided below.

In one embodiment, processing unit 4 is a small, portable device forexample worn on the user's wrist or stored within a user's pocket. Theprocessing unit may for example be the size and form factor of acellular telephone, though it may be other shapes and sizes in furtherexamples. The processing unit 4 may include much of the computing powerused to operate head mounted display device 2. In embodiments, theprocessing unit 4 communicates wirelessly (e.g., WiFi, Bluetooth,infra-red, or other wireless communication means) to one or more hubcomputing systems 12.

FIGS. 2 and 3 show perspective and side views of the head mounteddisplay device 2. FIG. 3 shows the right side of head mounted displaydevice 2, including a portion of the device having temple 102 and nosebridge 104. Built into nose bridge 104 is a microphone 110 for recordingsounds and transmitting that audio data to processing unit 4, asdescribed below. At the front of head mounted display device 2 isroom-facing video camera 112 that can capture video and still images.Those images are transmitted to processing unit 4, as described below.

A portion of the frame of head mounted display device 2 will surround adisplay (that includes one or more lenses). In order to show thecomponents of head mounted display device 2, a portion of the framesurrounding the display is not depicted. The display includes alight-guide optical element 115, opacity filter 114, see-through lens116 and see-through lens 118. In one embodiment, opacity filter 114 isbehind and aligned with see-through lens 116, light-guide opticalelement 115 is behind and aligned with opacity filter 114, andsee-through lens 118 is behind and aligned with light-guide opticalelement 115. See-through lenses 116 and 118 are standard lenses used ineye glasses and can be made to any prescription (including noprescription). In one embodiment, see-through lenses 116 and 118 can bereplaced by a variable prescription lens. In some embodiments, headmounted display device 2 will include one see-through lens or nosee-through lenses. In another alternative, a prescription lens can goinside light-guide optical element 115. Opacity filter 114 filters outnatural light (either on a per pixel basis or uniformly) to enhance thecontrast of the virtual imagery. Light-guide optical element 115channels artificial light to the eye. More details of opacity filter 114and light-guide optical element 115 are provided below.

Mounted to or inside temple 102 is an image source, which (in oneembodiment) includes microdisplay 120 for projecting a virtual image andlens 122 for directing images from microdisplay 120 into light-guideoptical element 115. In one embodiment, lens 122 is a collimating lens.

Control circuits 136 provide various electronics that support the othercomponents of head mounted display device 2. More details of controlcircuits 136 are provided below with respect to FIG. 4. Inside ormounted to temple 102 are ear phones 130, inertial measurement unit 132and temperature sensor 138. In one embodiment shown in FIG. 4, theinertial measurement unit 132 (or IMU 132) includes inertial sensorssuch as a three axis magnetometer 132A, three axis gyro 132B and threeaxis accelerometer 132C. The inertial measurement unit 132 sensesposition, orientation, and sudden accelerations of head mounted displaydevice 2. The inertial measurement unit 132 may sense pitch, roll andyaw (of the users' head, for example). The IMU 132 may include otherinertial sensors in addition to or instead of magnetometer 132A, gyro132B and accelerometer 132C.

Microdisplay 120 projects an image through lens 122. There are differentimage generation technologies that can be used to implement microdisplay120. For example, microdisplay 120 can be implemented in using atransmissive projection technology where the light source is modulatedby optically active material, backlit with white light. Thesetechnologies are usually implemented using LCD type displays withpowerful backlights and high optical energy densities. Microdisplay 120can also be implemented using a reflective technology for which externallight is reflected and modulated by an optically active material. Theillumination is forward lit by either a white source or RGB source,depending on the technology. Digital light processing (DLP), liquidcrystal on silicon (LCoS) and Mirasol® display technology from Qualcomm,Inc. are examples of reflective technologies which are efficient as mostenergy is reflected away from the modulated structure and may be used inthe present system. Additionally, microdisplay 120 can be implementedusing an emissive technology where light is generated by the display.For example, a PicoP™ display engine from Microvision, Inc. emits alaser signal with a micro mirror steering either onto a tiny screen thatacts as a transmissive element or beamed directly into the eye (e.g.,laser).

Light-guide optical element 115 transmits light from microdisplay 120 tothe eye 140 of the user wearing head mounted display device 2.Light-guide optical element 115 also allows light from in front of thehead mounted display device 2 to be transmitted through light-guideoptical element 115 to eye 140, as depicted by arrow 142, therebyallowing the user to have an actual direct view of the space in front ofhead mounted display device 2 in addition to receiving a virtual imagefrom microdisplay 120. Thus, the walls of light-guide optical element115 are see-through. Light-guide optical element 115 includes a firstreflecting surface 124 (e.g., a mirror or other surface). Light frommicrodisplay 120 passes through lens 122 and becomes incident onreflecting surface 124. The reflecting surface 124 reflects the incidentlight from the microdisplay 120 such that light is trapped inside aplanar substrate comprising light-guide optical element 115 by internalreflection. After several reflections off the surfaces of the substrate,the trapped light waves reach an array of selectively reflectingsurfaces 126. Note that one of the five surfaces is labeled 126 toprevent over-crowding of the drawing. Reflecting surfaces 126 couple thelight waves incident upon those reflecting surfaces out of the substrateinto the eye 140 of the user.

As different light rays will travel and bounce off the inside of thesubstrate at different angles, the different rays will hit the variousreflecting surfaces 126 at different angles. Therefore, different lightrays will be reflected out of the substrate by different ones of thereflecting surfaces. The selection of which light rays will be reflectedout of the substrate by which surface 126 is engineered by selecting anappropriate angle of the surfaces 126. In one embodiment, each eye willhave its own light-guide optical element 115. When the head mounteddisplay device 2 has two light-guide optical elements, each eye can haveits own microdisplay 120 that can display the same image in both eyes ordifferent images in the two eyes. In another embodiment, there can beone light-guide optical element which reflects light into both eyes.

Opacity filter 114, which is aligned with light-guide optical element115, selectively blocks natural light, either uniformly or on aper-pixel basis, from passing through light-guide optical element 115.However, in general, an embodiment of the opacity filter 114 can be asee-through LCD panel, an electrochromic film, or similar device whichis capable of serving as an opacity filter. Opacity filter 114 caninclude a dense grid of pixels, where the light transmissivity of eachpixel is individually controllable between minimum and maximumtransmissivities. While a transmissivity range of 0-100% is ideal, morelimited ranges are also acceptable, such as for example about 50% to 90%per pixel, up to the resolution of the LCD.

A mask of alpha values can be used from a rendering pipeline, afterz-buffering with proxies for real-world objects. When the system rendersa scene for the augmented reality display, it takes note of whichreal-world objects are in front of which virtual objects as explainedbelow. If a virtual object is in front of a real-world object, then theopacity may be on for the coverage area of the virtual object. If thevirtual object is (virtually) behind a real-world object, then theopacity may be off, as well as any color for that pixel, so the userwill see the real-world object for that corresponding area (a pixel ormore in size) of real light. Coverage would be on a pixel-by-pixelbasis, so the system could handle the case of part of a virtual objectbeing in front of a real-world object, part of the virtual object beingbehind the real-world object, and part of the virtual object beingcoincident with the real-world object. Displays capable of going from 0%to 100% opacity at low cost, power, and weight are the most desirablefor this use. Moreover, the opacity filter can be rendered in color,such as with a color LCD or with other displays such as organic LEDs, toprovide a wide FOV.

Head mounted display device 2 also includes a system for tracking theposition of the user's eyes. For example, head mounted display device 2includes eye tracking assembly 134 (FIG. 3), which has an eye trackingillumination device 134A and eye tracking camera 134B (FIG. 4). In oneembodiment, eye tracking illumination device 134A includes one or moreinfrared (IR) emitters, which emit IR light toward the eye. Eye trackingcamera 134B includes one or more cameras that sense the reflected IRlight. The position of the pupil can be identified by known imagingtechniques which detect the reflection of the cornea. Such a techniquecan locate a position of the center of the eye relative to the trackingcamera. Generally, eye tracking involves obtaining an image of the eyeand using computer vision techniques to determine the location of thepupil within the eye socket. In one embodiment, it is sufficient totrack the location of one eye since the eyes usually move in unison.However, it is possible to track each eye separately.

In one embodiment, the system will use four IR LEDs and four IR photodetectors in rectangular arrangement so that there is one IR LED and IRphoto detector at each corner of the lens of head mounted display device2. Light from the LEDs reflect off the eyes. The amount of infraredlight detected at each of the four IR photo detectors determines thepupil direction. That is, the amount of white versus black in the eyewill determine the amount of light reflected off the eye for thatparticular photo detector. Thus, the photo detector will have a measureof the amount of white or black in the eye. From the four samples, thesystem can determine the direction of the eye.

Another alternative is to use four infrared LEDs as discussed above, butone infrared CCD on the side of the lens of head mounted display device2. The CCD will use a small mirror and/or lens (fish eye) such that theCCD can image up to 75% of the visible eye from the glasses frame. TheCCD will then sense an image and use computer vision to find the image,much like as discussed above. Thus, although FIG. 3 shows one assemblywith one IR transmitter, the structure of FIG. 3 can be adjusted to havefour IR transmitters and/or four IR sensors. More or less than four IRtransmitters and/or four IR sensors can also be used.

Another embodiment for tracking the direction of the eyes is based oncharge tracking. This concept is based on the observation that a retinacarries a measurable positive charge and the cornea has a negativecharge. Sensors are mounted by the user's ears (near earphones 130) todetect the electrical potential while the eyes move around andeffectively read out what the eyes are doing in real time. Otherembodiments for tracking eyes can also be used.

FIG. 3 shows half of the head mounted display device 2. A full headmounted display device would include another set of see-through lenses,another opacity filter, another light-guide optical element, anothermicrodisplay 120, another lens 122, room-facing camera, eye trackingassembly, micro display, earphones, and temperature sensor.

FIG. 4 is a block diagram depicting the various components of headmounted display device 2. FIG. 5 is a block diagram describing thevarious components of processing unit 4. Head mounted display device 2,the components of which are depicted in FIG. 4, is used to provide amixed reality experience to the user by fusing one or more virtualimages seamlessly with the user's view of the real world. Additionally,the head mounted display device components of FIG. 4 include manysensors that track various conditions. Head mounted display device 2will receive instructions about the virtual image from processing unit 4and will provide the sensor information back to processing unit 4.Processing unit 4, the components of which are depicted in FIG. 4, willreceive the sensory information from head mounted display device 2.Based on that exchange of information and data, processing unit 4 willdetermine where and when to provide a virtual image to the user and sendinstructions accordingly to the head mounted display device of FIG. 4.

Some of the components of FIG. 4 (e.g., room-facing camera 112, eyetracking camera 134B, microdisplay 120, opacity filter 114, eye trackingillumination 134A, earphones 130, and temperature sensor 138) are shownin shadow to indicate that there are two of each of those devices, onefor the left side and one for the right side of head mounted displaydevice 2. FIG. 4 shows the control circuit 200 in communication with thepower management circuit 202. Control circuit 200 includes processor210, memory controller 212 in communication with memory 214 (e.g.,D-RAM), camera interface 216, camera buffer 218, display driver 220,display formatter 222, timing generator 226, display out interface 228,and display in interface 230.

In one embodiment, the components of control circuit 200 are incommunication with each other via dedicated lines or one or more buses.In another embodiment, the components of control circuit 200 is incommunication with processor 210. Camera interface 216 provides aninterface to the two room-facing cameras 112 and stores images receivedfrom the room-facing cameras in camera buffer 218. Display driver 220will drive microdisplay 120. Display formatter 222 provides information,about the virtual image being displayed on microdisplay 120, to opacitycontrol circuit 224, which controls opacity filter 114. Timing generator226 is used to provide timing data for the system. Display out interface228 is a buffer for providing images from room-facing cameras 112 to theprocessing unit 4. Display in interface 230 is a buffer for receivingimages such as a virtual image to be displayed on microdisplay 120.Display out interface 228 and display in interface 230 communicate withband interface 232 which is an interface to processing unit 4.

Power management circuit 202 includes voltage regulator 234, eyetracking illumination driver 236, audio DAC and amplifier 238,microphone preamplifier and audio ADC 240, temperature sensor interface242 and clock generator 244. Voltage regulator 234 receives power fromprocessing unit 4 via band interface 232 and provides that power to theother components of head mounted display device 2. Eye trackingillumination driver 236 provides the IR light source for eye trackingillumination 134A, as described above. Audio DAC and amplifier 238output audio information to the earphones 130. Microphone preamplifierand audio ADC 240 provides an interface for microphone 110. Temperaturesensor interface 242 is an interface for temperature sensor 138. Powermanagement circuit 202 also provides power and receives data back fromthree axis magnetometer 132A, three axis gyro 132B and three axisaccelerometer 132C.

FIG. 5 is a block diagram describing the various components ofprocessing unit 4. FIG. 5 shows control circuit 304 in communicationwith power management circuit 306. Control circuit 304 includes acentral processing unit (CPU) 320, graphics processing unit (GPU) 322,cache 324, RAM 326, memory controller 328 in communication with memory330 (e.g., D-RAM), flash memory controller 332 in communication withflash memory 334 (or other type of non-volatile storage), display outbuffer 336 in communication with head mounted display device 2 via bandinterface 302 and band interface 232, display in buffer 338 incommunication with head mounted display device 2 via band interface 302and band interface 232, microphone interface 340 in communication withan external microphone connector 342 for connecting to a microphone, PCIexpress interface for connecting to a wireless communication device 346,and USB port(s) 348. In one embodiment, wireless communication device346 can include a Wi-Fi enabled communication device, BlueToothcommunication device, infrared communication device, etc. The USB portcan be used to dock the processing unit 4 to hub computing system 12 inorder to load data or software onto processing unit 4, as well as chargeprocessing unit 4. In one embodiment, CPU 320 and GPU 322 are the mainworkhorses for determining where, when and how to insert virtualthree-dimensional objects into the view of the user. More details areprovided below.

Power management circuit 306 includes clock generator 360, analog todigital converter 362, battery charger 364, voltage regulator 366, headmounted display power source 376, and temperature sensor interface 372in communication with temperature sensor 374 (possibly located on thewrist band of processing unit 4). Analog to digital converter 362 isused to monitor the battery voltage, the temperature sensor and controlthe battery charging function. Voltage regulator 366 is in communicationwith battery 368 for supplying power to the system. Battery charger 364is used to charge battery 368 (via voltage regulator 366) upon receivingpower from charging jack 370. HMD power source 376 provides power to thehead mounted display device 2.

As noted in the discussion of FIG. 1C, two 3D rays are calculated in oneembodiment. The first 3D ray is based on the orientation of the user'shead. The second 3D ray may be a stabilized version of the first 3D ray.Also, the amount of stabilization may be inversely proportional to thevariance of the motion of the first 3D ray.

FIGS. 6A-6C are diagrams to help illustrate how the interaction ray 66can be stabilized in a way that is inversely proportional to variance ina first 3D ray, in one embodiment. FIG. 6A shows an example of a first3D ray 660 for four recent points in time t1-t4. Also depicted is theinteraction ray (second 3D ray) 66 for the same four points in timet1-t4. Typically, the variance may be determined for many more recentpoints in time.

The first 3D ray 660 is represented by the longer rays. Note that thedifference in length is merely to be able to distinguish between thefirst and second rays. This example corresponds to rotation of theobject about the z-axis. This results in a component of motion for the3D vectors in the x-y plane.

When the variance in the angles with respect to rotation about an axisis high, very little or no stabilizing of the interaction ray 66 isapplied, in one embodiment. In this example, the variance in the anglesof rotation about the z-axis is sufficiently high such that nostabilizing is applied to the motion of the first 3D ray 660 in the x-yplane. Thus, the interaction ray 66 essentially mirrors the first 3D ray660 for each of the four points in time. For example, the interactionray 66 closely tracks the first 3D ray 660.

For some embodiments, the motion of the first 3D ray 660 could beconsidered to be about the z-axis. Thus, this could be considered to bestabilizing the rotation of the 3D ray about the z-axis, at least forsome cases. Note that stabilizing is performed separately for rotationabout different axes, in one embodiment.

FIGS. 6B and 6C shows an example in which variance of the angle ofrotation about an axis is low. Therefore, the amount of stabilizing ofthe interaction ray 66 is high. FIG. 6B shows a first 3D ray for ninerecent points in time. FIG. 6C shows an interaction ray 66 for the samenine points in time. The length of the arrows is used to representdifferent points in time. The shortest arrows in each diagram is for t1and the longest for t9. This example corresponds to rotation of theobject about the z-axis. In this example, both the first 3D ray 660 andthe interaction ray 66 have their origin at the origin of the xyzcoordinate system. Note that this may also be true for the example ofFIG. 6A.

FIG. 6B shows that the first 3D ray 660 first drifts a little to theright and then drifts back to the left. That is, first there is someclockwise rotation about the z-axis, and then there is somecounter-clockwise rotation about the z-axis. However, the variance ofthe angle of rotation about the z-axis for the recent time period isrelatively low compared to the example of FIG. 6A.

FIG. 6C shows the second 3D ray or interaction ray 66 that may resultfrom the low variance in FIG. 6B. In this example, the stabilizingresults in the interaction ray 66 being constant (the length of thearrows simply represent later points in time). Thus, if the user's headis moving slightly to the right and then slightly to the left, thiscould indicate small variance of the angle with respect to rotationabout an axis defined by their spine (e.g., z-axis). Applying highstabilizing factor creates stability in the interaction ray 66, despitesome motion of the user's head. Therefore, the user has better controlover the interaction ray 66. Note that all rays in FIGS. 6A-6C are shownin the x-y plane to simplify the drawings.

FIG. 7 is a flowchart of one embodiment for determining an instabilityfactor based on variance in an Euler angle. The instability factor maybe used to control motion of the interaction ray 66. Thus, theinstability factor may be used to control how closely the interactionray 66 tracks the first 3D ray. The process may be performed for one ofthe Euler angles, such as pitch, roll, or yaw. The following discussioncould apply to any of the Euler angles. Step 701 is to access HMD sensordata. The use of this data is described below.

In step 702, the present Euler angle is determined. Sensor data from theHMD 2 may be used to help determine the Euler angle. As mentioned above,one possibility is to determine a face unit vector by defining a planeof the user's face, and taking a vector perpendicular to that plane. Theface vector may be referred to as an x-vector, with reference to theHMD-coordinate system of FIG. 1A. Furthermore, it is possible todetermine head vectors (or axes) in the z-direction and the y-direction.For example, the y-direction head vector could roughly connect theperson's ears. For example, the z-direction head vector could roughlystart at the middle of the person's head and be perpendicular to theother two head vectors. Other possibilities exist. Theses head vectorsmay be identified from the camera image data returned from the cameras112 on head mounted display device 2. In particular, based on what thecameras 112 on head mounted display device 2 see, the associatedprocessing unit 4 is able to determine the head vectors representing auser's head orientation.

Note that the position and orientation of a user's head may also oralternatively be determined from analysis of the position andorientation of the user's head from an earlier time (either earlier inthe frame or from a prior frame), and then using the inertialinformation from the IMU 132 to update the position and orientation of auser's head. Information from the IMU 132 may provide accurate kinematicdata for a user's head, but the IMU typically does not provide absoluteposition information regarding a user's head. This absolute positioninformation, also referred to as “ground truth,” may be provided fromthe image data obtained from cameras on the head mounted display device2 for the subject user and/or from the head mounted display device(s) 2of other users.

The Euler angle may be determined in reference to the HMD-basedcoordinate space (see FIG. 1A, 1B). Note that origin of the HMD-basedcoordinate space could be somewhere in the person's head. Thus, Eulerangles may represent rotations of the head about the axes of theHMD-based coordinate space. For example, the angles may represent arotation of the head about the x-axis by an angle α, a rotation of thehead about the y-axis by an angle β, and a rotation of the head aboutthe z-axis, by an angle γ. Note that it is not required that all of theangles be determined.

In step 704, the present Euler angle is recorded. Note that Euler anglesfor previous times are presumed to have been previously recorded. As oneexample, Euler angles for the last second may be stored. Note that step704 may discard older values, such that only the most recent Euler angleare recorded. The sampling rate could range considerably, depending onthe application and considerations such as computational time available,as well as desired accuracy.

In step 706, the present Euler angle is compared with the stored valuesto determine how much the Euler angle has changed or varied over therecent time period.

In step 708, the variance of the Euler angle over the recent time periodis characterized. In one embodiment, the largest delta, the smallestdelta, and the mean delta are determined. The largest delta refers tothe largest difference between the present Euler angle and one of therecorded values. The smallest delta refers to the smallest differencebetween the present Euler angle and one of the recorded values. The meandelta refers to the average difference between the present Euler angleand each of the recorded values. Thus, the variance may be relative tothe present Euler angle, but other possibilities exist.

Other measures could be used to characterize the variance of the Eulerangle over the recent time period. As noted above, the variance of theEuler angle over time could be calculated as the mathematical variance.However, the term variance as it is used in step 708 is not limited to astrict mathematical definition of variance.

In step 710, an instability factor is determined for the Euler angle.This instability factor may be for the recent time period for which theEuler angles are recorded, as discussed above. In one embodiment, theinstability factor is based on a mean average of the differences betweenthe Euler angle for the present time and the Euler angles with respectto rotation about the axis of the head over the recent time period.

In one embodiment, step 710 calculates the difference between thelargest delta and the smallest delta from step 608. This is referred toas the delta variance. An instability factor is then determined inaccordance to Equation 1:

Instability=(DeltaVar*VarWeight)+(DeltaMean*MeanWeight)  (1)

In Equation 1, DeltaVar refers to the difference between the largestdelta and the smallest delta, and VarWeight refers to a weightingfactor. DeltaMean refers to the mean delta determined in step 608, andMeanWeight refers to a weighting factor. Suitable values for VarWeightand MeanWeight may be determined empirically. The example of Equation 1is just one example of many possibilities.

There are many variations to the process of FIG. 7. In one embodiment,step 706 is modified by comparing a fixed value for the Euler angle withthe recorded values, as opposed to comparing the present value of theEuler angle with the recorded values for the recent time period.

After the instability factor is determined, it may be applied to theinteraction ray 66 to stabilize it. FIG. 8 is a flowchart of oneembodiment of a process of applying an instability factor to determine anew interaction ray 66. Step 802 is to access a present position of the3D ray. This may include an origin and direction for the first 3D ray.The 3D ray may be a 3D vector having an origin at, for example, theorigin of the HMD-based coordinate system.

Step 804 is to access the more recent (or last) position of theinteraction ray 66. This may include an origin and direction for theinteraction ray. The interaction ray 66 may have the same origin as thefirst 3D ray, but that is not required. The interaction ray 66 may be a3D vector having an origin at, for example, the origin of the HMD-basedcoordinate system.

Step 906 is to modify the position of the interaction ray 66 based onthe present position of the 3D ray and the instability factor. In oneembodiment, the system alters how closely the interaction ray 66 tracksthe 3D ray based on the variance of the Euler angle over time, thecloseness of the tracking being inversely proportional to the varianceof the Euler angle.

In one embodiment, the system stabilizes the interaction ray 66 when theinstability factor indicates that the Euler angle is relatively stable,but allows the interaction ray 66 to freely track the 3D ray when theinstability factor indicates that the Euler angle is relativelyinstable.

FIG. 9 is a flowchart of one embodiment in which a head curser isstabilized based on variance of angle of rotation of the user's head.

In step 902, a holographic image is presented in an HMD 2. As oneexample to help illustrate, a newspaper article is displayed. In oneembodiment, the holographic image is kept is place in the real world tohelp the user interact with it. For example, the holographic image ismade to appear as if it is on a table or wall. Of course, the user couldmove the holographic image if desired.

In step 904, rotation of a user's head is tracked with respect to one ormore axes. These axes are defined relative to the head itself, in oneembodiment. Step 904 may include tracking pitch and yaw, as examples. Inone embodiment, the z-axis corresponds roughly to line through the skullthat is an extension of the user's spine. Thus, rotation about thez-axis corresponds to the user rotating their head such that they lookleft or right. This may be referred to as yaw.

In one embodiment, the y-axis corresponds roughly to a line thatconnects the user's ears. Thus, rotation about the y-axis corresponds tothe user moving their head up and down such as a nodding motion. Thismay be referred to as pitch.

It is also possible to track rotation about an x-axis. In this example,the x-axis may correspond roughly to interaction ray 66. In anapplication such as a cursor, tracking this rotation about the x-axismight not be needed.

In step 904, the position and orientation of a user's head may bedetermined, at least in part based on a camera image data returned fromthe cameras 112 on head mounted display device 2.

In step 906, a first 3D vector 660 is determined based on theorientation of the user's head. In one embodiment, the origin is at apoint midway between the user's eyes. The vector extends “straight out”from the user's face in the direction in which they are facing. Onepossibility is for this to be the x-axis previously mentioned. Onepossibility is to define a plane for the user's face and project a lineperpendicular to that face plane. This face plane may be identified bydetermining a position of a user's eyes, nose, mouth, ears or otherfacial features. The face plane may be identified from the camera imagedata returned from the cameras 112 on head mounted display device 2.

In step 908, the system determines variance of rotation of the user'shead about one or more of the axes.

In step 910, the instability factor is used to generate the interactionray 66. In one embodiment, the process of FIG. 8 is used. Thus, theinteraction ray 66 may be generated by stabilizing motion of the first3D vector 660. The amount of dampening is inversely proportional to thevariance of rotation of the user's head, in one embodiment. A separateamount of stabilization may be applied for each axis. As one example, afirst amount of stabilization is applied for pitch and a second amountof stabilization is applied for yaw.

In step 912, the position of the interaction ray 66 is determined withrespect to the holographic image 60. The system (e.g., HMD 2) maydetermine or otherwise access the apparent x, y and z positions of allelements in the virtual image at the current time. By “apparent” x, y,and z position it is meant the position in the real world at which theimage appears to be. The system can determine whether the interactionray (e.g., a second 3D vector) 66 intersects any point in the virtualimage.

In step 914, the system determines whether a selection of an elementthat the interaction ray 66 is colliding with is received. The selectioncould be a voice command, etc.

In step 916, the system takes some action in response to the selection.The action might be to response to selection of a hyperlink, a menuselection, etc. The system might highlight the selected element. Theselection might be the user manipulating virtual controls on the virtualimage to pause, rewind, fast-forward, change the volume or change thecontent of a displayed video. Many other possibilities exist.

Other variations are possible of the process of FIG. 9. For example,rather than have the interaction ray 66 be a cursor, the interaction ray66 might be a pointer. This pointer might be used to point at an objectin the real world instead of an element in a holographic image.

Note that in the foregoing example, pitch, roll, and yaw were used. Therotation about an axis of the head could be a rotation about an axisother than pitch, roll, or yaw.

In one embodiment, stabilizing is provided for translation of the head.This stabilizing may be in addition to stabilizing based on rotation ofthe head. Translation of the head could potentially cause motion in theinteraction ray 66. One example is that breathing might cause an up anddown motion that is not necessarily reflected in rotation about an axis.

FIG. 10 is a flowchart of one embodiment in which stabilizing isprovided for variance of translation of the user's head. Translationinvolves movement of the origin of the HMD-based coordinate system inone embodiment. Note that the virtual image in the HMD 2 does notnecessarily move when the user's head translates. One possible reasonfor this is that the virtual image can be fixed in place in the realworld, such as by making it appear to be on a desk or wall.

In step 1002, translation of the head is tracked using a sensor. Thiscould be based on any of the sensors in the HMD 2. Note that translationof the head might cause movement in the first 3D ray 660. As oneexample, the origin of the first 3D ray could move. Moving the origin ofthe first 3D ray might cause the entire ray to move a similar amount.For example, when the origin moves up 3 centimeters, the entire 3D raymight move up 3 centimeters, depending on how the 3D ray is formed.Although possibilities exist.

Step 1002, may include recording recent values for a position of theoriginal of the HMD-based coordinate system. This, in turn, may bedetermined based on the position of the user's head. In one embodiment,the position of the user's head is determined by determining the 3Dposition of the HMD 2. This could be based on any of the sensors in theHMD 2, of even sensors outside of the HMD such as a camera. Note thatmore than one sensor could be used, such as using IMU to determinechanges in the position of the HMD 2.

In step 1004, variance of the translation of the head is determined. Instep 1006, motion of the first 3D ray 660 that is due to the translationis stabilized. The stabilization may be inversely proportional to thevariance of the translation of the head, in one embodiment.

Step 1006, may include determining a stabilization factor based just onthe variance of the translation of the object. A separate stabilizationfactor may be determined for whatever axes are being analyzed, aspreviously described. All of the various stabilization factors can beapplied to the interaction ray 66.

In general, any combination of the stabilization factors can be appliedto the interaction ray 66. FIG. 11 is a flowchart showing that manycombination of the stabilization factors can be applied to theinteraction ray 66. If stabilization for variance in pitch is to beapplied, then an instability factor for variance in pitch is determinedand applied in step 1102. In one embodiment, the first 3D ray is avector that has an origin (e.g., at the origin of the HMD-basedcoordinate system). The first 3D ray may be characterized by pitch andyaw, and roll with respect to, for example, the HMD-based coordinatesystem. The second 3D ray may also be a vector that has an origin (e.g.,at the origin of the HMD-based coordinate system). The second 3D ray mayalso be characterized by pitch, yaw, and roll with respect to, forexample, the HMD-based coordinate system. Step 1102 may control howclosely the pitch of second 3D vector track the pitch of the first 3Dvector based on variance in the pitch of the head. Stated another way,step 1102 may control how closely the pitch of second 3D vector tracksthe pitch of the first 3D vector based on variance in the pitch of thefirst 3D vector.

If stabilization for variance in yaw is to be applied, then aninstability factor for variance in yaw is determined and applied in step1104. Step 1104 may control how closely the yaw of second 3D vectortracks the yaw of the first 3D vector based on variance in the yaw ofthe head. Stated another way, step 1104 may control how closely the yawof second 3D vector tracks the yaw of the first 3D vector based onvariance in the yaw of the first 3D vector.

If stabilization for variance in roll is to be applied, then aninstability factor for variance in roll is determined and applied instep 1106. In one embodiment, the first 3D vector has a roll componentassociated with it. This roll component is in addition to the origin anddirection of the 3D vector. Step 1106 may control how closely roll ofsecond 3D vector tracks the roll of the first 3D vector based onvariance in the roll of the head. Stated another way, step 1106 maycontrol how closely the roll of second 3D vector tracks the roll of thefirst 3D vector based on variance in the roll of the first 3D vector.

If stabilization for variance in translation of the head is to beapplied, then an instability factor for variance in translation isdetermined and applied in step 1108. Step 1108 may control how closelythe origin of second 3D vector tracks the origin of the first 3D vectorbased on variance in the translation of the head.

In one embodiment, at least one of steps 1102, 1104, or 1106 is applied.In one embodiment, at least one of steps 1102 or 1104 is applied.However, any combination of steps 1102, 1104, 1106, and 1108 arepossible.

Aspects of the present disclosure are described herein with reference toflowchart illustrations, sequence diagrams and/or block diagrams ofmethods, apparatuses (systems) and computer program products accordingto embodiments of the disclosure. It will be understood that each blockof the flowchart illustrations and/or block diagrams, and combinationsof blocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. Similarly, each arrow of asequence diagram may likewise be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor of a general purpose computer (or computing device), specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart, sequence diagram and/or block diagram blockor blocks.

The storage device and working memory are examples of tangible,non-transitory computer- or processor-readable storage devices. Storagedevices include volatile and nonvolatile, removable and non-removabledevices implemented in any method or technology for storage ofinformation such as computer readable instructions, data structures,program modules or other data. Computer storage devices include RAM,ROM, EEPROM, cache, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical disk storage, memorysticks or cards, magnetic cassettes, magnetic tape, a media drive, ahard disk, magnetic disk storage or other magnetic storage devices, orany other device which can be used to store the desired information andwhich can accessed by a computer. Computer storage devices do notinclude propagated signals.

Some embodiments have been described herein as being implanted asinstructions performed by a processor. Alternatively, or in addition,embodiments described herein can be performed, at least in part, by oneor more hardware logic components. For example, and without limitation,illustrative types of hardware logic components that can be used includeField-programmable Gate Arrays (FPGAs), Application-Specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-Chip systems (SoCs), Complex Programmable Logic Devices(CPLDs), etc.

One embodiment disclosed herein includes apparatus comprising a sensorand logic that monitors orientation of a person's head using the sensor,including monitoring rotation about an axis of the head, includingrecording an Euler angle with respect to rotation about the axis of thehead. The logic determines a three-dimensional (3D) ray based on theorientation of the head. The 3D ray has a motion that may preciselytracks the Euler angle over time. The logic generates an interaction raythat tracks the 3D ray to some extent. The logic determines a varianceof the Euler angle over a recent time period. The logic stabilizes theinteraction ray based on the variance of the Euler angle over the recenttime period despite some rotation about the axis of the head. An amountof stabilizing may be inversely proportional to the variance of theEuler angle. The logic determines a collision of the interaction raywith a 3D coordinate.

The apparatus of the above paragraph may further comprise a near-eye,see-through display. The logic may present a holographic image on thenear-eye, see-through display, the logic determines a collision of theinteraction ray with respect to the holographic image.

In one embodiment of the apparatus of either of the two precedingparagraphs, the 3D ray is a first 3D vector that originates from thehead and the interaction ray is a second 3D vector that originates fromthe head.

In one embodiment of the apparatus of either of the three precedingparagraphs calculates an instability factor for the Euler angle over therecent time period. The instability factor may be a function ofdifferences between an Euler angle for the present time and the Eulerangle with respect to rotation about the axis of the head for differentpoints in time over the recent time period. The logic modifies theinteraction ray based on the instability factor and a present positionof the 3D ray. The interaction ray may be stabilized when the varianceof the Euler angle over the recent time period is low despite somemotion of the head about the axis. The interaction ray closely tracksthe 3D ray when the variance of the Euler angle over the recent timeperiod is high. In one embodiment, the instability factor is based on amean average of the differences between the Euler angle for the presenttime and the Euler angles with respect to rotation about the axis of thehead over the recent time period.

In one embodiment of the apparatus of the preceding paragraphs, thefirst Euler angle tracks pitch of the head and the second Euler angletracks yaw of the head.

In one embodiment of the apparatus of the preceding paragraphs the logicfurther tracks translation of the person's head. The logic determines avariance of the translation of the head over the recent time period. Thelogic may alter how closely the interaction ray tracks the 3D ray basedon the variance of the Euler angle over the recent time period and thevariance of the translation of the head over the recent time period. Thelogic may stabilize the interaction ray based on the variance of thetranslation of the head over the recent time period despite sometranslation of the head during the recent time period.

One embodiment includes method comprising the following. Headorientation is tracked using a sensor, including tracking rotation aboutan axis of the head. Values for an angle of rotation about the axis ofthe head over a recent time interval are recorded. A firstthree-dimensional (3D) ray based on actual orientation of the head isdetermined. The first 3D ray has a motion that tracks the actualorientation of the head over time. The first 3D ray has a directcorrespondence of the rotation about the axis of the head. A variance ofthe recorded angle over the recent time interval is determined. A second3D ray is determined based on actual position of the first 3D ray andthe variance of the recorded angle over the recent time interval. Thismay include stabilizing motion of the second 3D ray to a degree that isinversely proportional to the variance. A collision of the second 3D raywith a 3D coordinate is determined.

One embodiment includes an apparatus comprising a sensor; a see-through,near-eye display; processor readable storage having instructions storedthereon; and a processor coupled to the sensor, the processor readablestorage, and the see-through, near-eye display. The instructions whichwhen executed on the processor cause the processor to do the following.The processor presents a holographic image on the near-eye, see-throughdisplay. The holographic image may be associated with points in 3D spacein a field of view of the see-through, near-eye display. The processoraccesses data from the sensor. The processor tracks head orientationusing the sensor data, and tracks pitch and yaw of the head orientation.The processor determines a first variance in the pitch over time and asecond variance in the yaw over time. The processor determines a first3D vector based on the head orientation. The first 3D vector has anorigin at a point on the head. The first 3D vector has a pitch and a yawthat track the pitch and a yaw of the head over time. The processorgenerates a second 3D vector that has a pitch and a yaw that tracks thefirst 3D vector. The processor controls how closely the pitch and theyaw of second 3D vector track the pitch and the yaw of the first 3Dvector based on the first variance in the pitch of the head and thesecond variance in the yaw of the head. The processor tracks the pitchof the second 3D vector proportional to the first variance andstabilizes the pitch of the second 3D vector inversely proportional tothe first variance. The processor tracks the yaw of the second 3D vectorproportional to the second variance and stabilizes the yaw of the second3D vector inversely proportional to the second variance. The processordetermines a collision of the second 3D vector with a 3D point that isassociated with the holographic image.

Support should also be provided for multiple dependent claims byproviding the equivalent description in this section, for example, bystating that the embodiment described in the previous paragraph may alsobe combined with one or more of the specifically disclosed alternatives

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims. It is intended that the scopeof the invention be defined by the claims appended hereto.

What is claimed is:
 1. An apparatus comprising: a sensor; and logicconfigured to monitor orientation of a person's head based on data fromthe sensor, including the logic being configured to monitor rotationabout an axis of the person's head and to determine an angle withrespect to rotation about the axis of the person's head; the logicconfigured to determine a first three-dimensional (3D) ray based on theorientation of the person's head, the first 3D ray having a motion thattracks the angle over time; the logic configured to generate a second 3Dray based on the first 3D ray; the logic configured to determine avariance of the angle over a time period, the variance being based ondifferences between the angle for a first time in the time period andthe angle for different points in time over the time period; the logicconfigured to stabilize the second 3D ray based on the variance of theangle over the time period, an amount of stabilizing being inverselyproportional to the variance of the angle; and the logic configured todetermine an intersection of the second 3D ray with a real world 3Dcoordinate.
 2. The apparatus of claim 1, wherein the angle is an Eulerangle.
 3. The apparatus of claim 1, wherein the logic is configured tohave the second 3D ray track the first 3D ray to an extent that dependson the variance.
 4. The apparatus of claim 1, wherein the logic isconfigured to stabilize the second 3D ray based on the variance of theangle over the time period despite some rotation about the axis of theperson's head.
 5. The apparatus of claim 1, wherein the first 3D ray isa first 3D vector that originates from the person's head, the second 3Dray is a second 3D vector that originates from the person's head andtracks the first 3D vector to at least some extent.
 6. The apparatus ofclaim 1, wherein the logic is configured to stabilize the second 3D raywhen the variance of the angle over the time period is low despite somerotation of the person's head about the axis, and to have the second 3Dray closely track the first 3D ray when the variance of the angle overthe time period is high.
 7. The apparatus of claim 1, wherein thevariance is based on a mean average of the differences between the anglefor the first time and the angles with respect to rotation about theaxis of the person's head over the time period.
 8. The apparatus ofclaim 1, wherein the variance is a mathematical variance.
 9. Theapparatus of claim 1, further comprising: a near-eye, see-throughdisplay; wherein the logic is configured to present a holographic imagein the near-eye, see-through display to appear to be at the real world3D coordinate, wherein when logic determines that the second 3D rayintersects the real world 3D coordinate the logic determines that thesecond 3D ray intersects the holographic image.
 10. A method comprising:tracking a person's head orientation using a sensor, including trackingrotation about an axis of the head; recording values for an angle ofrotation about the axis of the head over a time interval; determining athree-dimensional (3D) vector based on actual orientation of the head,the 3D vector has a motion that tracks the orientation of the head overtime, the 3D vector has a direct correspondence to the rotation aboutthe axis of the head; determining a variance of the recorded angle overthe time interval based on differences between the recorded angle for afirst time in the time interval and the recorded angle for differentpoints in time over the time interval; generating an interaction vectorbased on actual position of the 3D vector and the variance of therecorded angle over the time interval, including stabilizing motion ofthe interaction vector to a degree that is inversely proportional to thevariance; and determining an intersection of the interaction vector witha real world 3D coordinate.
 11. The method of claim 10, wherein theangle is an Euler angle.
 12. The method of claim 10, wherein thetracking a person's head orientation includes: tracking pitch in thehead orientation, the angle of rotation about the axis of the head isbased on the pitch of the head orientation.
 13. The method of claim 10,wherein the tracking a person's head orientation includes: tracking yawin the head orientation, the angle of rotation about the axis of thehead is based on the yaw of the head orientation.
 14. The method ofclaim 13, wherein the tracking a person's head orientation furtherincludes tracking pitch in the head orientation, and further comprising:determining a variance of the pitch over the time interval based ondifferences between the pitch for the first time in the time intervaland the pitch for different points in time over the time interval,wherein the stabilizing motion of the interaction vector is furtherinversely proportional to the variance of the pitch over the timeinterval.
 15. The method of claim 10, wherein the variance is amathematical variance.
 16. An apparatus comprising: a sensor; asee-through, near-eye display; processor readable storage havinginstructions stored thereon; and a processor coupled to the sensor, theprocessor readable storage, and the see-through, near-eye display, theinstructions which when executed on the processor cause the processorto: present a holographic image on the see-through, near-eye display,the holographic image being associated with points in 3D space in afield of view of the see-through, near-eye display; access data from thesensor; track pitch and yaw of a user's head orientation based on thesensor data; determine a pitch variance in the pitch over a recent timeperiod and a yaw variance in the yaw over the recent time period,comprising the instructions causing the processor to determine the pitchvariance based on differences between the pitch for the present time andthe pitch for different points in time over the recent time period andto determine the yaw variance based on differences between the yaw forthe present time and the yaw for different points in time over therecent time period; determine a 3D vector based on the head orientation,the 3D vector having an origin at a point on the head, the 3D vectorhaving a pitch and a yaw that track the pitch and the yaw of the headover time; generate an interaction vector that has a pitch and a yawthat tracks the 3D vector; control how closely the pitch and the yaw ofthe interaction vector track the pitch and the yaw of the 3D vectorbased on the pitch variance and the yaw variance, the instructions causethe processor to stabilize the pitch of the interaction vector inverselyproportional to the pitch variance, the instructions cause the processorto stabilize the yaw of the interaction vector inversely proportional tothe yaw variance; and determine an intersection of the interactionvector with a 3D point that is associated with the holographic image.17. The apparatus of claim 16, wherein the pitch variance is amathematical variance, the yaw variance is a mathematical variance. 18.The apparatus of claim 16, wherein the pitch variance is based on a meanaverage of the differences between the pitch for the present time andthe pitch for different points in time over the recent time period,wherein the yaw variance is based on a mean average of the differencesbetween the yaw for the present time and the yaw for different points intime over the recent time period.
 19. The apparatus of claim 16, whereinthe instructions that control how closely the pitch and the yaw of theinteraction vector track the pitch and the yaw of the 3D vector furthercause the processor to: calculate a pitch instability factor for thepitch of the head based on the pitch variance and a yaw instabilityfactor for the yaw of the head based on the yaw variance; stabilize theinteraction vector with respect to pitch when the pitch instabilityfactor indicates that the pitch variance in the pitch of the head islow; stabilize the interaction vector with respect to yaw when the yawinstability factor indicates that that the yaw variance in the yaw ofthe head is low; allow the interaction vector to freely track the 3Dvector with respect to pitch when the pitch instability factor indicatesthat the pitch variance in the pitch of the head is high; and allow theinteraction vector to freely track the 3D vector with respect to yawwhen the yaw instability factor indicates that the yaw variance in theyaw of the head is high.
 20. The apparatus of claim 16, wherein theinstructions further cause the processor to: track translation of thehead; determine a variance of the translation of the head over time; andcontrol how closely the origin of the interaction vector tracks anorigin of the 3D vector based on the variance of the translation of thehead, the origin of the interaction vector tracks the origin of the 3Dvector more closely when the variance of the translation of the head ishigh than when the variance of the translation of the head is low.